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What is claimed is: 

A system for interacting with displays and all devices that use 
such displays comprised of 

a. a display, 

a sensor or earner^ 



camera, 



b. 

c. 

d. 
e. 



a pointing device 



the pointing device and a corresponding, 



a method for det acting the pointing device, 
a method for est iblishing the mapping between the position of 
location on the display. 



1 le 



2. A system accorcjin 
in addition to registering the image of 
one of (i) the image of the display and 
device can produce on the display. 



that can be registered by the sensor or 



g to claim 1 wherein the sensor or camera, 
pointing object, can also register at least 
ii) the reflection or effect that the pointing 



3. A system as defined by claim 1 which commands the 
positioning of a pointing icon on the display. 



1 4. A system according to claim 1 wherein the pointing device is a 

2 part of the human body such as a hand or a finger, or an ornament or device worn on 

3 the human body such as a glove or thimble. 



1 5. A system according to claim 1 wherein the pointing device is 

2 used to point to regions of the display py way of changing its position, 

3 attitude, or presentation. 



1 
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6. A system according to claim 1 wherein the pointing device is 
used to define a particular point or region on the display. 
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1 7. A system accobding to claim 1 wherein the pointing device is 

2 used to define a vector on the plane M the display that indicates a direction 

3 and magnitude relative to or with reipect to an item on the display or a 

4 region of the display. / 

I 1 8. A system according to claim 3 wherein the pointing icon on 

2 the display can be registered by thq sensor or camera. 

1 9. A system according to claim 8 which also includes a method 

2 for correcting the offsets between di) the position of the pointing device, or reflection, 

3 or effect thereof on the display as observed by the user or by the sensor or the camera, 

4 and (ii) the position of the pointer fecon on the display. 

1 10, A system ap defined by claim 1 which also includes at least 

2 one of the following: I 

3 a. a method for selecting or highlighting a specific item or icon 

4 on the display, I 

5 b. a method for activating a specific process, program, or menu 

6 item represented on the display, and 

7 c, a method for writing, scribing, drawing, highlighting, 

8 annotating, or otherwise producing marks on the display. 

1 ^ method ttbr detecting the pointing device comprising 

2 a. retrieval oi data or image from a sensor or camera, and 

3 b. analysis on the data or image from the sensor or camera to 

4 locate the pointing device in the aata, or locating at least a set of the picture elements 

5 in the image that comprise the reridition of the pointing device. 
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12. A method according to claim 1 1 wherein the characteristics 
that distinguish the pointing devicelfrom other objects in the data from the 
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sensor or the image from the c amera are known a priori 



13. Amethbd 
that distinguish the pointing d 
the image from the camera are 
data acquired from the sensor 



according to claim 1 1 wherein the characteristics 
vice from other objects in the data from the sensor or 
determined based analysis of at least one set of the 
or one image acquired from the camera. 



14. Ameth([)d 
that distinguish the pointing device 
the data from the sensor or in 

a. acquirir 
two images from the camera, 
or the camera and one without 

b. comparing 



1 le 



one 



15. Amethbd 
modifications are made to the 
the camera pursuant the analy 
camera. 



according to claim 1 1 wherein adjustments or 
position, sensitivity, and other settings of the sensor or 
is of the data or image retrieved from the sensor or the 



16. AmethjDd 
procedures for the method is qarried 
mechanisms available on one 
camera, or the pointing devicd 
display, or the device producii l 



according to claim 13 wherein the characteristics 

from other objects, whose rendition are present in 
image from the camera, is obtained by 
at least two sets of data from the sensor or at least 

with the pointing device in view of the sensor 
and 

the two sets with one another. 



according to claim 1 1 wherein at least part of the 

out using at least in part the computing 
more of the following: the display, or the sensor or 
or the device producing the signal shown on the 
g the pointing icon on the display. 



A methqd for establishing the mapping between the set of 
positions that a pointing devic^ can take and the set of corresponding locations on the 
display comprising: 

a. defining the range of positions that the pointing device can 
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5 assiyne, 

6 \ b. defining the boundaries of the range of positions that the 

7 pointing device can take with geometric representations, 

8 \ c. transforming the geometric representation of the arrangement 

9 of regionsVtn the display so that it fits optimally into the boundaries of the range of 
1 0 positions thi^t the pointing device can take, 

1 \ 1 8. A method according to claim 1 7 wherein the range of positions 

2 that the pointing device may assume is defined by querying the user to point to a set 

3 of points on the display. 



1 19. \ A method according to claim 18 wherein the range of positions 

2 that the pointing de\^ce can assume is defined by the boundary contours of the 

3 display as they are registered by the sensor or the camera. 

1 20. i^method according to claim 19 wherein at least one special 

2 display image is used to establish the mapping between the positions that a pointing 

3 device can take and a corresponding locations on the display. 

1 2 1 . A memod according to claim 1 7 wherein at least part of the 

2 procedures for the method is darried out using at least in part the computing 

3 mechanisms available on one or more of the following: the display, or the sensor or 

4 camera, or the pointing device, qr the device producing the signal shown on the 

5 display, or the device producing me pointing icon on the display. 
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2. A method foAdetecting the display comprising 

a. retrieval of data or image from a sensor or camera, and 

b. analysis of the oata or image from the sensor or camera to 
locate the display in the data, or locatinaat least a set of the picture elements in 
the image that comprise the rendition of me display in the image. 
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23. A method 
that distinguish the display from 
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iccording to claim 22 wherein the characteristics 
Dther objects in the data from the sensor or the 



image from the camera are known a priori 



24. A methoc 
that distinguish the display fron 
image from the camera are dete 
acquired from the sensor or one 



according to claim 22 wherein the characteristics 
other objects in the data from the sensor or the 
mined based on analysis of at least one set of the data 
image acquired from the camera. 



25. A method according to claim 22 wherein the display refers to 



the range of positions that the 



lointing device can take. 



from 



26. A meth({>d 
that distinguish the display 
from the sensor or in the imag 

a. acquirin; 
two images from the camera, 
camera and one with the displ 

b. compailin 



27. A methbd 
modifications are made to the 
the camera pursuant the analy; 
camera. 



according to claim 24 wherein the characteristics 
other objects, whose rendition are present in the data 
from the camera, is obtained by 
g at least two sets of data from the sensor or at least 
with the display off in view of the sensor or the 
on, and 

g the two sets with one another. 



one 



ay 



according to claim 22 wherein adjustments or 
position, sensitivity, and other settings of the sensor or 
of the data or image retrieved from the sensor or the 
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1 28. A metho|l according to claim 22 wherein at least part of the 

2 procedures for the method is capied out using at least in part the computing 

3 mechanisms available on one on more of the following: the display, or the sensor or 

4 camera, or the pointing device, or the device producing the signal shown on the 
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A| display, or the device producing the pointing icon on the display. 





